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(57) Abstract 

A virtual microscope slide is created by using a computer-controlled microscope (10) to capture a plurality of low magnification 
images of a specimen which are tiled to create a reconstructed macro image (24). A plurality of higher magnification images are also 
captured and tiled to create a micro image (26). The macro (26) and micro (24) images are then stored, along with their mapping coordinates 
for later, interactive viewing. 
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METHOD AND APPARATUS FOR CREATING 
A VIRTUAL MICROSCOPE SLIDE 

Field of the Invention 

This is a Continuation- In-Part of prior patent 
5 application Serial No. 08/805,856, filed March 3, 1997, 
which is hereby incorporated by reference in its 
entirety. This invention relates to a method of, and an 
apparatus for, acquiring and constructing tiled digital 
images from a specimen on a support, such as a microscope 
10 slide, and for storing, and transferring the image for 
viewing by another at a local or remote location. 

Background of the Invention 

The invention described in the aforesaid 
application answers a need, a requirement to image and 

15 digitally record an object in a relatively flat plane at 
high resolution/magnification. Today, it is impractical 
to construct an optical image sensor large enough to 
cover the entire image area e.g., of a specimen on a 
microscope slide, at the required resolution. This is 

20 because lens size and resolution/magnification issues 

limit the size of the field of view of magnified objects 
and their resulting images. Viewing through a microscope 
is akin to viewing through a periscope in that one sees a 
very small field of view even at low magnifications, such 

25 as 1.25X. A pathologist using a microscope often scans a 
slide to obtain in his mind an overall view or sense of 
what constitutes the specimen and he remembers the 
general locations of the diagnostically significant, 
small pieces of the specimen. Usually, these are the 

3 0 diseased areas, such as malignant or potentially 

malignant portions of the specimen. To obtain higher 
resolution and magnification of these suspicious 
portions, the pathologist switches to a higher 
magnification objective lens but then the field of view 

35 becomes much smaller again. Often, the pathologist 
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switches back and forth between the lower magnification, 
larger field of view objective lens to orient himself 
relative to the specimen and the high magnification, 
smaller field of view to obtain the detailed, high 
5 resolution view of the suspicious area on the specimen. 
Thus, the user never receives a magnified, condensed 
overall view of the specimen or a portion of the specimen 
but must remember the series of views taken at low 
magnification. Likewise, at high resolution, high 

10 magnification, the user never receives or views a 

collection of adjacent images but must interrelate these 
successive images in the user's mind. 

A similar problem exists on the Internet or 
intranet where a pathologist may receive a single field 

15 of view magnified image taken from a specimen over the 

Internet or the intranet on his browser. The pathologist 
must be provided with explanations to coordinate the high 
resolution view with the lower resolution view. The 
number of views available to the pathologist is very 

20 limited, and the pathologist is unable to select other 

views or to scroll to neighboring views at the areas that 
are most interesting to the pathologist. 

In the aforesaid prior application, there is 
disclosed a method and apparatus whereby a person may 

25 construct a low magnification, digitized overall, image 
view of the entire specimen on a slide or a selected 
portion of the specimen on a slide, such as the basal 
layer of a tissue section. The overall, low 
magnification digitized image allows the user to 

3 0 understand where the user is presently located in his 
viewing and where the user may want to make the next 
observation. That is, the low magnification overall view 
is generally in color and provides to the experienced 
user a visual overall or thumbnail view of the slide and 

3 5 shows the possible areas of interest for malignancy or 
other diseases which manifest themselves at certain 
locations on the specimen image being viewed. This low 
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magnif ication overall view enables the user to select 
thereon the points of interest that the user wants to 
view at a higher magnification. 

The overall view was constructed by taking by a 
5 large number of low magnification images of the specimen 
through a microscopic scanning system and then coherently 
assembling and coordinating these respective smaller 
views or images (hereinafter "image tiles") into one 
coherent, low magnification, macro image from the 
10 specimen. Often the digitized macro image is reduced in 
size by a software system to even a smaller size to be 
displayed on a local screen or to be transferred over a 
low bandwidth or a high bandwidth channel to a remote 
viewing screen. 

15 The prior application teaches how to assemble a 

large number of image tiles, for example, 3 5 image tiles 
for the macro image, and then to take a series of other 
tiles of a higher magnification or magnifications which 
will also be viewed by the user. To this end, the user 

20 is provided with a marker, such as a cursor or the like, 
to select the defined area of interest, and by a simple 
command, to cause the selected, higher magnification 
digitized images to appear on the screen for viewing by 
the user. The higher magnification images may be one of 

25 several magnifications or resolutions such as 10X, 20X 
and 4 OX. 

As disclosed in the aforesaid application, it is 
preferred to allow the user, such as a pathologist, to 
quickly flip back and forth between the high resolution 

3 0 micro image and the low macro resolution image or to 

provide separate split screens whereby the pathologist is 
shown an overall macro view and a marker showing where 
the current higher magnification view is located. 
Because of the multiple magnifications, the user may 

35 change to an intermediate magnification such as would be 
accomplished by switching between intermediate objective 
lenses. This provides the pathologist with views which 
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correspond to changing back and forth between objective 
lenses in a microscope, a procedure with which most 
pathologists are familiar and have been trained. 

Additionally, the aforesaid application provides 
5 the user with a scrolling feature that allows the user to 
shift into the viewing screen adjacent, magnified images 
on the screen so that the pathologist is not limited to 
only seeing just a full tile view but may see adjacent 
image material from adjacent, neighboring tile images. 

10 In the aforesaid patent application, there is a 

disclosure of transmitting the low magnification image 
over a local area network or over the Internet through 
various servers and computers. The tiled images that 
were being transmitted were achieved by use of a fully 

15 computer controlled microscope which allowed the user to 
navigate along a specimen area of interest, such as along 
a basal area or to other suspicious points spread 
throughout the specimen to acquire tiled images of 
selected areas so that the entire specimen would not have 

20 to be digitized and stored. As disclosed in the 

preferred embodiment in the aforesaid application, an 
Internet browser remotely-controlled, automated 
microscope could be used by a pathologist from a remote 
location to view the reconstructed macro image tiles; 

2 5 and, with his manipulation of the microscope, using an 

intranet or Internet browser, could acquire single images 
at higher magnifications if desired. While several 
people could see the particular digitized images being 
transmitted out over the Internet as they were acquired 

3 0 by a particular pathologist and several people could view 

the stored images, there was still a problem of control 
at operation of the microscope by each person viewing the 
digitized images, and a problem with acquiring and 
transmitting large areas of higher magnification images 
35 using the tiling method. 

As stated above in greater detail, the current 
state of archiving the digital images achieved through a 
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microscope is often by having photographs or by video 
tapes. The photographs are difficult to use as is a 
video tape particularly when the user wants to move 
rapidly back and forth between various images and to 
5 scroll through various adjacent parts of the specimen 

image. Further, current archival methods lack an overall 
macro image of the specimen, which allows the user to 
know exactly where the particular high resolution view is 
from when it is making an analysis of the high resolution 
10 image. 

While digitized images can be stored 
magnetically or otherwise digitized and recorded on 
various recording mediums, no current archival system 
allows the user to toggle back and forth between high 

15 magnification images and low magnification images or 
between various images at different magnifications such 
as that achieved by a pathologist switching microscope 
objective lenses in real time to get the macro and micro 
images from the same location on the specimen. 

20 Heretofore, the practice of pathology has been relatively 
limited to the use of microscopes and to the pathologist 
having to use the microscope to review the particular 
specimen. 

There is a need for a dynamic system whereby one 
25 or more or several pathologists, including a consulting 
pathologist, may view the same area simultaneously and 
interact with one another either in diagnosis or in 
analysis. Also, it would be best if the images from the 
specimen could be stored so that a pathologist could 
3 0 easily examine the images at his leisure using an 

intranet or Internet browser at a later date merely by 
accessing the particular web site where the images are 
located. 

It will be appreciated that a host of problems 
35 need to be solved to allow Internet or intranet users to 
view on their respective monitors useful, low resolution, 
macro images and high resolution, micro images of several 
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adjacent, original microscope images. One of the first 
problems is how to seam together neighboring tile images 
to form a seamless overall view of these tiles. 
Heretofore, attempts to seam the tiles used software to 
5 combine the pixels at the tile boundaries and have been 
generally unsuccessful. Another problem is that of 
mapping of coordinates beginning with the coordinates, 
usually X and Y coordinates, from and at the microscope 
stage carrying the slide and then the mapping of 

10 coordinates on the scanning screen not only for one 

magnification but also to coordinate the mapping for the 
respective multiple resolution images taken typically at 
1.25X, 10X and 4 OX or more. These coordinates must be 
maintained for a large number of tiled images, e.g., 40 

15 tiled images for one macro image. In order for the 

remote user to view these tile images and to flip back 
and forth between different resolution, tiled images, the 
user's computer and monitor not only must receive the 
addresses and stored parameters for each pixel but must 

20 also run them on a generic viewing program. 

Another problem with acquiring image tiles and 
sending them over a low bandwidth Internet channel is 
that both the storage requirements on the server and the 
amount of data acquired per slide become high, such as 

25 for example, 120 megabytes to one gigabyte. The 120 

megabytes is only achieved by not taking image tiles of 
the entire specimen but only image tiles from the areas 
selected by the pathologist when tracing at high 
resolution along basal layers or only at the dispersed, 

3 0 suspicious cancer appearing area in a breast cancer. 

Even with this selective interaction by a pathologist in 
constructing the macro and micro digitized images with a 
vastly reduced amount of image tiles relative to that 
which would be acquired if the entire specimen where 

3 5 imaged at each of the multiple magnifications, the 
acquired amount of data is a monstrous problem of 
transmitting in a reasonable amount of time over a narrow 
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bandwidth channel to an ordinary web browser having 
limited storage capacity. While rough compression 
techniques could be used, they cannot be used at the 
expense of providing the high resolution image that the 
5 pathologist must have for diagnosis of the specimen. 



Summary of the Invention 

In accordance with the present invention, there 
is provided a new and improved method and apparatus for 
constructing digitally scanned images from a microscope 

10 specimen, for storing the digitally scanned images in a 
tiled format convenient for viewing without a microscope, 
and for transferring the tiled images at multiple 
magnifications for viewing by another at a remote 
location. This is achieved by assembling together 

15 several adjacent, original microscope views at a first 
magnification to obtain an overall macro view of the 
specimen and assembling together several adjacent 
original microscope views at a higher magnification to 
create a combined data structure. The data structure may 

2 0 then be transferred to the remote viewer to provide this 

viewer multiple resolution macro and micro images of 
areas on the slide specimen. The data structure is 
constructed by digitally scanning and storing the low 
magnification images with their mapping coordinates and 
25 likewise, digitally scanning and storing higher 

magnification images with their mapping coordinates. 
Further, a pathologist may interactively select only 
those diagnostically significant areas of the specimen 
for digital scanning and storing to reduce significantly 

3 0 the number of image pixels stored at high resolution. 

The data structure can be transmitted over the 
Internet or intranet to allow multiple users to consult 
on a particular microscope each using his own virtual 
images of the specimen. These users each may flip back 
3 5 and forth between different resolution images in a manner 
similar to that achieved when shifting among objective 
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lens for different resolution views. However, the 
preferred embodiment of this invention provides a marker 
on the overall macro view showing the remote user where 
the higher resolution image is located on the specimen so 
5 that the user does not have to remember the location of 
the higher resolution image. Unlike the single, small 
optical field of view currently available, the remote 
user is provided with a series of abutted, tiled images 
each being substantially equal to one small optical field 
10 of view. Thus, the remote user is provided with better 
and larger macro and micro tiled images than the single, 
small optical fields of view taken at the same 
magnifications of a single tiled image. 

The preferred data structure is also provided 
with a generic viewing program that allows the remote 
user to manipulate and interpret the tiled images on the 
user's browser. This generic viewing program is self- 
contained with its own display and the interpretative 
program is usable with a variety of computers, browsers 
and monitors. The data structure uses selectively 
compressed data to reduce the huge amount of acquired 
data, e.g. 120- megabytes, into a small amount of data, 
e.g. 1.4 megabytes. Such smaller, more manageable 
amounts of data can be transmitted over a low bandwidth 
25 channel such as the Internet without the loss of 
resolution that would interfere with the remote 
pathologist's analysis. Further, the interactive program 
allows the pathologist to scroll and to view neighboring 
image areas of neighboring image tiles which were 
3 0 currently unavailable to the pathologist until the 

inventions set forth in the aforesaid application and in 
this application. 

Turning now in greater detail to aspects of this 
invention, problems with achieving tileable (i.e. 
contiguous images which can be seamlessly abutted next to 
each other to recreate the original image, but at 
different magnifications) multiple images of a specimen 



20 



35 
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on a microscope slide are overcome by the system of the 
invention. The system includes a microscope and stage in 
which digital locations on the stage have been 
predetermined in accordance with an electromechanical 
5 addressable coordinate system (X-Y for convenience) . 
Each point on the stage is assigned an "X" and a " Y" 
coordinate which uniquely defines its location. The 
increments in each of the X and Y directions are 
established at a predefined amount for example in .1 

10 micrometer increments. A key factor in achieving 

superior resolution of the specimen images at higher 
magnifications is to establish many more physical 
increments on the stage for each pixel of the image 
sensor and of the intended display. For example, at 

15 1.25X magnification, 64 points on the stage correspond to 
one pixel on a CCD optical sensor, which corresponds to 
one pixel on a 64 0 by 480 monitor (for a VGA display) , 
using the bitmap addressing and scrollable image method 
described herein. 

2 0 Once the coordinate system is defined for the 

microscope stage, when a specimen on a microscope slide 
is placed on it, each feature of interest on the slide 
can be uniquely located with reference to the stage. 
Then the microscope system is used to digitally scan the 
25 image. The first scan is done at a relatively small 

magnification since this image will be used to provide a 
"macro" image of the entire specimen. In the preferred 
embodiment, 1.25X magnification is used. The microscope 
system then scans the slide using the 1.25X objective. 

3 0 Since the image is detected by rectangular optical 

sensors, such as the optical sensors in a CCD grid, the 
stage must be moved in relatively larger increments to 
place the next adjacent physical part of the slide 
exactly in the region where that rectangular area will be 
3 5 precisely imaged on the CCD sensor. 

Although the area traveled is relatively large, 
the precision must be high to enable alignment of the 



WO 98/39728 



PCTYUS98/04011 



-10- 

image parts within the pixel resolution of the CCD 
sensor. For example, at the 1.25X magnification, 48,143 
X steps and 35,800 Y steps are necessary to move the 
specimen object on the stage to a new, contiguous region 
5 for optical imaging on the CCD sensor. The signal 

produced by the optical sensors in the CCD grid are then 
transmitted to a computer which stores the image signals 
in a series of tiled images. Since each image frame is 
defined by predetermined X-Y coordinates, these can be 
10 easily converted into a series of contiguous tiled 
images . 

To view the scanned digital image on a monitor, 
the computer uses a method of reserving an image bitmap 
corresponding to the entire size of the tiled image, 
15 e.g., in this instance, 10 x 8, 1.25X magnified tiled 
images are acquired. This requires an image bitmap of 
7,520 x 3,840 in size, using a 752 x 480 pixel CCD 
sensor. Since the X-Y coordinates are known for each 
image tile, and thus for each pixel in each tile, the 

2 0 bitmap can be used to coordinate and display the stored 

image tiles to present a fused macro view of the image 
with one-to-one pixel correspondence of the screen pixels 
with the image pixels. Typically, the screen pixels are 
fewer in X-Y size than the macro tiled image, (that is, 
25 the entire image cannot be viewed on the monitor without 
some sort of image compression); and in this case, the 
macro tiled image is scrolled on the viewable window 
segment of the screen to maintain the one-to-one 
correspondence. An advantage of the one-to-one 

3 0 correspondence is that significant image detail is 

available to the user. Further, since the physical X,Y 
position on the specimen is known through the stage 
coordinate relationship to the image pixels, the tiled 
macro images can be used to locate regions, and move the 
3 5 stage to that region from collection of higher 
magnification tiled images. 
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Since the nature of optics, i.e. lenses, is that 
they provide a generally circular image with a sharp 
central region and with fuzziness around the periphery of 
the image, the microscope system is designed to step 
5 through the various locations on the slide in such a 

manner to scan only the high resolution image portion in 
the center of the optical image. The fuzzy outer regions 
are discarded. This also has the benefit of ensuring a 
high resolution image once the tiled images are 

10 reconstructed for viewing by a user on a monitor. 

After the macro image is completed, a trained 
professional, such as an examining pathologist, views the 
image of the specimen by viewing. the macro image and 
looking for areas of interest. In general, most specimen 

15 slides contain only a few small areas of diagnostic 
significance. The balance of the slide is generally 
empty or not significant. When the examining pathologist 
views the slide, some areas may have been previously 
marked in the regions of interest for viewing and 

20 analysis at higher magnifications. Once these regions 
are marked, the microscope is set to the desired higher 
magnification and then only the marked regions are 
scanned and stored. Alternatively, he may define new 
areas directly on the macro image. In either case, the 

25 regions are outlined using a pointing device, such as a 
mouse, directly on the viewing window displaying the 
macro image. As described above with respect to the 
1.25X images, since the stage has a predefined coordinate 
system, the scanned higher magnification image portions 

30 can be easily located with respect to the macro image, 
creating a series of micro images. 

The fact that a typical microscope specimen 
slide contains only limited information of interest and 
the ability of the system embodying the invention to 

35 accurately locate such regions enables the system to 

create a virtual microscope slide, i.e. a data structure 
which can be used in place of the actual specimen slides. 
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Th is advantageously enables multiple users to consult on 
a particular specimen. Additionally, because of the 
reduced size of the data structures, they can be viewed 
locally on a personal computer, transmitted over an 
5 intranet or via the Internet globally. The created data 
structures can be stored on a variety of storage or 
recording media: for example, on a server's hard disk, a 
Jazz drive, a CD-Rom or the like. Storing the data 
structure on a portable storage media further enables the 

10 transfer and archiving of the microscopic slide data 
structures by multiple users. 

Another feature of the invention is a self- 
executing data structure. This is achieved by packaging 
the tiled images with an active, dynamic control program. 

15 When an active dynamic control program is used by a 

viewing program such as a common web browser, the browser 
can interpret the dynamic control program. This allows 
the user to interact and control the viewed images seen 
on the viewer's screen from the recording medium. More 

20 specifically, in the preferred embodiment of the 

invention, a large number of low magnification, digitized 
tiled images are formed and embedded in a data structure 
with linking information allowing them to be coherently 
tiled to each other during viewing to form a macro image, 

25 and a series of higher magnification tiled images also 
similarly constructed into a micro image, and a control 
program such as a JAVA applet, is provided and 
transferred with the macro and micro tiled images for use 
by a remote user. Thus, for example, the macro and micro 

30 tiled images with their active control program may be 

transmitted over an Internet or an intranet to a browser, 
or other application program for viewing the images, 
where the user may then access the browser to analyze the 
images at multiple resolutions and with a macro field of 

35 view before the user. This enables the viewing of the 
images in a manner similar to the use of an optical 
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microscope, but in this case visually the view is of a 
virtual microscope slide at multiple resolutions. 

Also, in accordance with the invention, the 
constructed, tiled macro and tiled micro images along 
5 with the control program can be placed on a web server 
and can be accessed locally and over a wide area, even 
globally, by multiple users at various times. For 
instance, a large number of previously scanned and 
recorded specimen slides, such as 300 specimen slides, 

10 may have their respective micro and macro tiled images 
put on a server. Medical students or pathology students 
then may each access the slide or all of the 300 slides 
and review them on their respective web browsers at their 
leisure. Likewise, a pathologist may dial up or 

15 otherwise connect through an internet service provider to 
the Internet or other long- level network and access a web 
server and obtain a particular patient's specimen 
results. Those results would have been stored as a data 
structure (including macro and micro tiled images along 

20 with the control and interpretative program) . The 

pathologist then may and perform an analysis at his home 
or in his office without needing to have or to control a 
microscope or the particular slide. The pathologist may 
toggle back and forth between the micro and macro images, 

25 and then dictate or otherwise prepare his analysis, 
findings or diagnosis from these stored images. This 
advantageously enables the pathologist to perform part, of 
his job in the convenience of his home or office and also 
enables a laboratory to maintain actual specimen slides 

30 in a safe and secure location, away from the potential of 
damage and without the necessity of shipping the slides 
for microscopic examination at a remote location. 

The control program, which in the preferred 
embodiment of the invention is a dynamic self -executing 

35 program such as a JAVA applet, allows the user to 

manipulate and interpret the images while on a browser. 
The dynamic, self -executing program is completely self- 



WO 98/39728 



PCT/US98/04011 



-14- 



15 



contained with its own display and interpretative program 
for operation by the user of the browser. 

The present invention is not limited to use on a 
browser since the tiled, digitized images and the active, 
5 control program may be stored on a CD-ROM or other 
portable storage medium and sent through the mail, or 
otherwise transferred to the user for review at the 
user's convenience with dedicated viewers. 

Thus, from the foregoing, it will be seen that 
10 there is provided a new and improved method of and 

apparatus for archiving of microscopic slide information 
on a storage medium with an active control program, which 
allows the display and interpretation of various micro 
and macro images . 

In accordance with another important aspect of 
the invention, there is provided with the self -executing 
data structure (the stored macro images, micro images and 
dynamic, self -executing program for viewing, 
reconstructing and manipulating the stored images) the 
ability to scroll through the displayed images. This 
allows the user not only to see one image tile at a 
particular magnification, but also to use a pointer or to 
otherwise move a point to cause displayed images from 
adjacent neighboring image tiles which were not 
25 previously viewable to be included in the field being 
viewed by the user. That is, the user may shift the 
viewing location across tile boundaries from one tile to 
another, and up or down, or right or left or to other 
points of interest in a normal two-dimensional scrolling 
30 manner. Thus, the user is provided with an archived 
stored slide at multiple magnifications which can be 
readily scrolled through in any arbitrarily chosen 
direction or directions. As in the aforesaid 
application, the user interactively will go to various 
areas of selected interest and operate a pointer or a 
marker to select for high magnification viewing the 



20 



35 
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particular area of interest and also do a scrolling of 
neighboring areas of interest. 

In addition to the Internet browser, the data 
images can be viewed, reconstructed and manipulated using 
5 a dynamic, self -executing program such as, for example, a 
JAVA applet or an ACTIVE-X applet. An advantage of using 
a dynamic, self -executing program which is linked with 
the data images on a data structure is that the data 
images can be viewed, reconstructed and manipulated 

10 independent of the operating system of the users 
computer. Additionally, the user does not have to 
acquire the latest version of the dynamic, self -executing 
program since it is already linked with and provided with 
the data images on the data structure or on the storage 

15 medium. Thus, the user can always view the data images, 
regardless of different program versions. 

The dynamic, self -executing program permits 
interchanging the image in its entirety simulating the 
visual effect of changing objectives in a regular, 

20 mechanical optical microscope view. Thus, the user can 
easily switch from one magnification to another and 
scroll through portions of the image, simulating tracking 
the image by moving the slide under the microscope lens. 
The dynamic, self -executing program permits 

25 scrolling the image in a window to enable viewing of the 
reconstructed large field of view images. The user can 
use a mouse, or other pointing device, to select a 
portion of the image on the large field of view image and 
the program will display that selected portion in another 

30 window at the desired magnification. 

A method of constructing a record of the digital 
image of a specimen on a microscope slide using image 
tiles includes scanning the image at a first low 
magnification so that substantially all of the specimen 

35 is obtained. Then the specimen is scanned at a second 
higher magnification so that images of selected (or all) 
sub-portions of the specimen are obtained. The spatial 
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relationships of the first lower magnification image to 
the second higher magnification images is used to 
reconstruct the image during viewing. The individual, 
sub-portions or tiles of the scanned image are seamed 
5 together by the dynamic, self -executing program to create 
a digital image of substantially larger areas than 
individually acquired image fields of view without 
tiling. 

A data structure according to the invention is 

10 created by first digitally scanning the desired specimen 
at a plurality of image magnifications. The scanned 
images are then stored in a series of contiguous image 
tiles. Then the stored images are linked with a dynamic, 
self -executing program. The data structure can be 

15 created using a software program. Images are preferably 
first stored as bitmap files (.bmp). (Note that storing 
the resulting image files in the bitmap format is 
different from the bit mapping method of creating the 
image files described herein.) An image compression 

20 program is used to convert the bitmap files to a JPEG 
(•jP9) format, which requires less storage space and 
consequently less time to display on a computer. The 
person creating the data structure can select how much 
detail to include in the conversion. JPEG images can be 

25 created for example, using 20 to 80% compression ratios 
of the original image. An advantage of the JPEG format 
is that essentially empty tiles (tiles with mostly white 
or black space) compress down to very small files. 
Detailed files, however, do not compress as much. 

30 Additionally, the dynamic, self -executing program may 

include compression algorithms for displaying the entire 
image or portions thereof in the viewing window. 

After downloading or installation of a data 
structure on a storage medium, when the user desires to 

35 view the data images, he uses a mouse and "clicks" on the 
icon for the self -executing data structure. The dynamic, 
self-executing program displays the image in a window. 
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Typically, the program will display a macro or thumbnail 
view of the entire specimen image at a lower 
magnification and a smaller window containing a 
particular image tile or groups of tiles at a higher 
5 magnification. The program enables the user to use the 
mouse or other pointing device to select a point or 
outline a region on the thumbnail view. The selected 
view will then be displayed in the smaller window at the 
second magnification. The user can move, the mouse or 

10 pointing device and the image in the smaller window will 
scroll with the selection on the thumbnail view. In this 
way, the program simulates movement of a microscope slide 
under the field of view of the mechanical microscope. 
However, it should be noted that because of the 

15 one-to-one correspondence between the CCD pixels and the 
screen pixels, not all macro images may be able to be 
displayed on the monitor. The user may scroll through 
the macro image or select a compression feature to 
display the entire macro image in the window. 

20 Another feature of the self -executing data 

structure is that when the image is displayed on the 
viewing screen, the user can select an image tile or sub- 
portion of the image and alternately view that portion of 
the image at each scanned magnification. For example, if 

25 the data was scanned at magnifications of 1.25X, 20X and 
40X, the user can "click" and see the same tile at each 
of those magnifications alternately. 

Brief Description of the Drawings 

FIG. 1 is a block diagram of a system according 
3 0 to the invention for creating and transmitting locally, 
over an intranet or via the Internet data structures of 
an image of specimen on a microscope slide; 

FIG. 1A is representation of a microscope slide 
which has been arbitrarily assigned to be scanned into 
35 eighty tiled images; 
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FIG IB is a representation of the detected 
signals of the individual pixel sensors in a CCD optical 
array after detecting a selected image area to tile and 
the referenced data files containing the information 
5 describing the detected signals; 

FIG. 2 is a screen view of a system embodying 
the present invention showing a low magnification image 
of a specimen on a microscope slide in one window, a high 
magnification image of a portion of the low magnification 
10 image selected by a region marker and a control window; 

FIG. 3 is a view of a display screen of the 
apparatus embodying the present invention showing the 
control window a low magnification window having a 
plurality of high magnification micro image regions 
15 delineated therein and a high magnification window 
including one or more of the micro image regions; 

FIG. 4 is a view of a macro image of an actual 
breast cancer specimen displayed at 1.25X as seen on a 
computer monitor; 
20 FIG. 5 is a view of the grid portion of FIG. 4 

outlining a region of interest selected by a pathologist 
displayed at 40X magnification; 

FIG. 6 is a block diagram of the steps in the 
mapping of the scanned image from the optical sensor 
25 array to computer bit map in memory to the display on a 
user's monitor; 

FIG. 7A is a file listing such as would be seen 
under Windows 95 file manager showing the data files 
included in a data structure for a breast cancer 
30 specimen; 

FIG. 7B is a file listing of a Java applet for 
controlling a data structure; 

FIG. 8 is file listing such as would be seen 
under Windows 95 file manager showing the data files 
3 5 included in an alternate data structure for a breast 
cancer specimen; 
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FIGS. 9A and 9B are a block diagram of the 
apparatus embodying the present invention; 

FIG. 10 is a block diagram of a portion of the 
apparatus shown in FIG. 9 showing details of a mechanical 
5 arrangement of a microscope; 

FIG. 11 is a flow diagram related to operation 
of the apparatus; 

FIG. 12 is a flow diagram of details of one of 
the steps in FIG. 11; 
10 FIG. 13 is a display screen showing control 

parameters to be manipulated thereon; 

FIG. 14 is a flow chart for a region outlying 

routine; 

FIG. 15 is a flow chart for a scanning and 
15 analyzing routine; 

FIG. 16 is a schematic showing of the limits of 
travel of the microscope stage with respect to the image 
tiles; 

FIG. 16A is a perspective view of the microscope 
20 stage and stepper motors and encoders providing a closed 
loop drive for the motors; 

FIG. 17 is a block diagram of a networked system 
allowing multiple workstations to obtain access to the 
microscope and to manipulate the microscope locally at 
25 each workstation; 

FIG. 17A is a view of the system described in 
connection with FIG. 10; and 

FIG. 18 is a block diagram of a remote networked 
system for distributing and accessing diagnostic images 
3 0 and data, i.e. virtual microscope slides, through a 
hypertext transport protocol based server directly or 
over a packet network. 

Detailed Description of the Preferred Embodiment 

FIG. 1 is a block diagram of a system according 
3 5 to the invention for creating, and transmitting over an . 
intranet or via the Internet a virtual microscope slide, 
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i.e. interrelated data structures and display procedures 
depicting at multiple resolutions, images of a specimen 
on a microscope slide. The system includes a microscope 
with a digital platform for supporting the microscope 
5 slide. Digital platform or stage 11 has been specially 
calibrated to include a large number of increments for 
locating portions of specimen images with high precision. 
After calibration and initial registration of stage 11 in 
the microscope setup, a microscope slide or other 
10 substrate with a specimen to be scanned is placed on 
stage 11. 

For exemplary purposes, the creation of virtual 
microscope slide specimen according to the invention will 
be described with respect to a breast cancer specimen. 

15 The first step in creating a data structure according to 
the invention is to establish a macro image of the entire 
specimen (or that portion of the specimen desired to be 
stored as the macro image) . The purpose for creating the 
macro or large area thumbnail image is to enable the 

20 viewer to see the entire specimen at once and to use the 
entire image to choose those significant portions thereon 
for viewing at greater magnification. In this example, 
the user has selected 1.25X as the magnification to 
display the entire breast cancer slide. Once specimen 

25 13a has been placed on stage 11, rotating optical 
assembly 15 are rotated to select lens 17 which 
corresponds to the 1.25X magnification. 

In accordance with the teachings of the prior 
patent application, the computer controlled microscope is 

3 0 moved to scan the entire image of specimen 13a. The 

focusing system is programmed to step through increments 
which detect/select only the high resolution center area 
of the field of view in order to avoid storing the fuzzy 
areas at the periphery of the field of view. In this 

35 example, the macro image will be stored in a 10 by 8 

array, for a total of 80 contiguous image tiles, as shown 
in FIG. 1A. 
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A typical microscope slide is about 77mm by 
25mm, where the usable area, without including the label, 
is about 57mm by 25m. Each of the 80 image segments is 
about 4.8mm by 3.5mm in dimension. This means each of 

5 the 80 image segments will be scanned separately and 
stored as a separate image tile. 

The precision of the microscope systems is set 
up so that each step of the motor has a precision of .1 
micron (micrometer) . In this example, the microscope is 

10 set up to move 48,143 steps in the X direction and 35,800 
steps in the Y direction at 1.25X magnification for each 
of the 80 image areas . At higher magnifications, the 
image areas to scan are considerably smaller, so the 
number of steps is corresponding smaller. For each of 

15 the 80 image areas, the microscope lens will detect only 
the high resolution center area of the field of view. 

The optical image of the desired image area is 
then detected by optical array sensor 19 (preferably a 
CCD sensor array) . In this example, each of the 80 

20 scanned areas is sensed by the entire array, which 

includes 752 pixels by 480 pixels. The optical array 
sensor sends electrical signals indicative of the ■ 
detected image to microscope controlled computer 32. 
Computer 32 stores the scanned images, including the top 

25 left X-Y stage coordinates for each of the 80 individual 
areas of the microscope slide. Each of the 80 scanned 
image areas' pixel locations are stored in a bit mapped 
file (i.e., a file which contains a map of the location 
of each bit in the area) which corresponds to the layout 

3 0 of the individual images thereon. Thus, all of the 

pixels from the image tile derived from region A on FIG. 
1A, which is the seventh from the left and in the top 
row, are individually assigned unique locations in the 
computer memory's bit-mapped file (FIG. 6), and are also 

35 stored in the data structure image tile file as shown in 
FIG. IB. 
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Each of the stored data image tiles is a 
standard image file with extension.bmp, and is of the 
order of one megabyte, i.e. each of the 752 x 480 pixels 
is stored as 3 bytes of red, green and blue image data 
5 (752 x 480 x 32 * 1,082,880 bytes) . Since the location 
of each image tile is known according to the bitmap, the 
complete microscope image can be recreated by painting 
(displaying) each image tile in accordance with its grid 
location. It should be noted 

10 To display the resulting image, computer 32 

calculates the appropriate portion to be displayed from 
each image tile depending upon the relative size of the 
display screen. Since the stored image data is usually 
greater than the size of the typical monitor, the viewer 

15 must scroll through the image on the window to view it 

entirely. However, an optional compression algorithm can 
be used to compress the entire image into the viewing 
window. The X-Y coordinate information is used by the 
viewing and manipulation program to reconstruct the image 

20 tiles into a complete image of the specimen. The 

resulting image is larger, and with better resolution 
than would be achieved if optics technology were able to 
construct a single lens capable of viewing the entire 
specimen in one field of view. In this example, each of 

25 the 80 image tiles has digital resolution of 752 x 480 
pixels, with corresponding optical resolution of 
approximate .2 microns at 40X to approximately 6.4 
microns at 1.25X. 

After the macro or thumbnail images are 

30 digitally scanned and stored with their X-Y coordinate 
information, the user then examines the macro image or 
original specimen for significant details. Typically, 
the user will highlight with a marking pen the areas to 
be viewed at higher magnification. The user then changes 

35 the magnification of optics system 15 to the desired 

higher magnification, moves the scanning system to bring 
the selected region into view. Computer 32 then repeats 
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the scanning and image tile creation process for the 
selected region, but at higher magnification and with a 
new grid system to locate the scanned selected regions. 

In the example, the user has selected region B 
5 shown on FIG. 1A to perform a second view at a higher 
magnification. The user selects, for example, a 40X 
magnification. The computer calculates the number of 
tiles to cover the selected area at 40X magnification and 
sets up a second grid. 

10 It should be noted that region B crosses over 

several of the larger tiles in FIG. 1A. Because of the 
extreme precision of the instrument, 0.1 micron 
resolution, locating such selected regions with high 
resolution is readily accomplished. As noted above, the 

15 compute calculates the size of the image portion, in this 
case as an example, X = 1500 and Y = 1200 stepping 
increments. Each image portion at the 40X resolution is 
detected by the optical sensor array, 752 by 480 pixels. 
Each resulting data file is stored in a separate, high 

20 magnification mapped area of memory so that the computer 
can easily recall the location of region B, or any of its 
200 individual image tiles, when requested by a user. 

Once the user has completed selecting and having 
the computer controlled microscope system scan and store 

25 the digital images in image tiles, computer 32 stores the 
mapped .brnp files along with their coordinate information 
and creates slide image data structure 31 in FIG. 1. 
Slide image data structure includes all of the bitmap 
image tile files at both magnifications (note that 

30 similarly, additional images could be stored at further 
magnifications, if desired), as well as X-Y coordinate 
information for the location of the various image tiles. 

FIG, 7A is a file listing such as would be seen 
under a Windows 95 file manager showing the data files 

35 included in a data structure for a breast cancer 

specimen. Included in the file listing are FinalScan.ini 
and SlideScan.ini as well as sixty bitmap data files. 
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Slidescan.ini is a listing of all the original bitmap 
(.bmp) files. The bitmap files represent the individual 
image tiles in the scan at, say, 1.25X magnifications, 
Slidescan.ini is set forth below in Table 1 and describes 
5 the X-Y coordinates for each image tile file. When the 
data structure is viewed by a control program, the 
program uses the X-Y coordinates to display all the image 
tiles contiguously. 
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TABLE 1 Slidescan.ini 





[Header] 


[Ssl7] 




x=278000 


x=133571 




y=142500 


y=35100 


5 


lXStepSize=48143 


[Ssl8] 




lYStepSize=35800 


x=181714 




iScannedCount =3 7 


y=35100 




[Ssl] 


[Ssl9] 




x=181714 


x=229857 


10 


y=142500 


y=35100 




[Sb2] 


[Ss20] 




x=133571 


x=278000 




y=142500 


y=-700 




[Ss3] 


[Ss21] 


15 


x=37285 


x=229857 




y=106700 


y=-700 




[Ss4] 


[Ss22] 




x=85428 


x=181714 




y=106700 


y=-700 


20 


[Ss5] 


[Ss23] 




x=133571 


x=133571 




y=106700 


y=-700 




[Ss6] 


[Ss24] 




x=181714 


x=85428 


25 


y=106700 


y=-700 




[Ss7] 


[Ss25] 




x=229857 


x=37285 




y=106700 


y=-700 




[Ss8] 


[Ss26] 


30 


x=229857 


x=-10858 




y=70900 


y=-700 




[Ss9] 


[Ss27] 




x=181714 


x=-10858 




y=70900 


y=-3650O 


35 


[SslO] 


[Ss28] 




x=133571 


x=37285 




y=70900 


y=-36500 




[Ssll] 


[Ss29] 




x=85428 


x=85428 


40 


y=70900 


y=-36500 




[Ssl2] 






x=37285 






y=70900 






[Ssl3] 




45 


x=-10858 






y=70900 






[Ssl4] 






x=-10858 






y=35100 




50 


[Ssl5] 






x=37285 






y=35100 






[Ssl6] 






X=85428 




55 


y=35100 
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[Ss30] 

x=133571 

y=-36500 

[Ss31] 
5 x=181714 

y=-36500 

[Ss32] 

x=229857 

y=-36500 
10 [Ss33] 

x=278000 

y=-36500 

[Ss34] 

x=278000 
15 y=-72300 

[Ss35] 

x=229857 

y=-72300 

[Ss36] 
20 x=181714 

y=-72300 

[Ss37] 

x=133571 

y=-72300 

25 Table 2 is a listing of the file FinalScan.ini, which is 
a listing the X-Y coordinates of the high magnification 
image tiles scanned and stored. 



TABLE 2 - FinalScan.ini 



[Header] 
30 tPatientID=mda027 
tAccession= 
tOperatorID= jwb 

tTimeOfScan=8/4/97 1:19:56 
PM 

35 lXStageRef =278000 

lYStageRef =142500 

iImageWidth=752 

i ImageHe ight =4 80 

lXStepSize=1590 
40 lYStepSize=1190 

lXOf f set=-1900 

lYOf f set=-400 

dMagnif icat ion=4 0 

lAnalysisImageCount=105 
45 lCalibrationImageCount=0 

[DaO] 

x=214532 
y=65584 
[Dal] 
50 x=212996 



y=65584 

[Da2] 
x=211460 
y=65584 

[Da3] 
x=209924 
y=65584 

[Da4] 
X=208388 
y=65584 

[Da5] 
x=206852 
y=65584 

[Da6] 
x=205316 
y=65584 

[Da7] 
x=203780 
y=65584 

[Da8] 
x=214532 
y=64400 
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[Da9] 
x=212996 
y=64400 

[DalO] 
5 x=211460 
y=64400 

[Dall] 
x=209924 
y=64400 
10 [Dal2] 

x=208388 
y=64400 

[Dal3] 
x=206852 
15 y=64400 

[Dal4] 
x=205316 
y=64400 

[Dal5] 
20 x=203780 
y=64400 

[Dal6] 
x=214532 
y=63216 
25 [Dal7] 
x=212996 
y=63216 

[Dal8] 
x=211460 
30 y=63216 

[Dal9] 
x=209924 
y=63216 

[Da20] 
35 X=208388 
y=63216 

tDa21] 
x=206852 
y=63216 
40 [Da22] 
x=205316 
y=63216 

[Da23] 
x=203780 
45 y=63216 

[Da24] 
x=214532 
y=62032 

[Da25] 
50 x=212996 
y=62032 

[Da26] 
x=211460 
y=62032 
55 [Da27] 
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X=209924 
y=62032 

[Da28] 
x=208388 
y=62032 

[Da29] 
X=206852 
y=62032 

[Da30] 
x=205316 
y=62032 

[Da31] 
X=203780 
y=62032 

[Da32] 
x=214532 
y=60848 

[Da33] 
x=212996 
y=60848 

[Da34] 
x=211460 
y=60848 

[Da35] 
x=209924 
y=60848 

[Da36] 
x=208388 
y=60848 

[Da37] 
x=206852 
y=60848 

[Da38] 
x=205316 
y=60848 

[Da39] 
x=203780 
y=60848 

[Da40] 
x=214532 
y=59664 

[Da41] 
x=212996 
y=59664 

[Da42] 
x=211460 
y=59664 

[Da43] 
x=209924 
y=59664 

[Da44] 
x=208388 
y=59664 

[Da45] 
X=206852 
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Computer 32 can also use the scanned image files 
to create a self -executing data structure. By 
compressing the .bmp images to .jpg and adding a dynamic, 
self -executing program which enables the user to view, 
5 reconstruct and manipulate the image tiles, the user can 
use the data structure as a virtual microscope slide of 
the original specimen. Preferably, the dynamic, self- 
executing program is a Java applet, such as shown on FIG. 
7B. 

■10 Computer 32 can provide the slide image data 

structure 31 directly or via an intranet browser 33 to 
local viewer 34, or via an Internet server 38. Slide 
image data structure 37 is shown as being directly 
accessible from Internet server 38. Alternatively, a 

15 user can download the slide image data structure on his 
own computer 39, use an internet browser 43 and view the 
reconstructed images. Another alternative is for 
computer 32 to store the slide image data structure on a 
CD-rom, Jazz drive or other storage medium. 

20 To view slide image data structure 31 or 37, the 

user, who for example, has acquired the data structure 
via a CD-rom, first installs the CD-rom in the CD-rom 
drive of his computer. Then the user opens up a browser 
or other applications program which can read the Java 

25 applet installed on the CD-rom with the image tiles. 

Note that in some instances no separate browser program 
may be required. In some case, the CD-rom may include 
the complete applications program for viewing, 
reconstructing and manipulating the image tiles. In the 

30 instant example, the user will then select the icon or 
file listing for the slide image data structure and the 
control program will display the data files. 

FIG. 2 is a screen view of a system embodying 
the present invention showing a low magnification image 

35 24 of a specimen on a microscope slide in one window, a 
high magnification image 26 of a portion of the low 
magnification image selected by a region marker 30 and a 
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control window 28. FIG. 3 is a view of a display screen 
of the apparatus embodying the present invention showing 
the control window 28, a low magnification window 24 
having a plurality of high magnification micro image 
5 regions 310 delineated therein and a high magnification 
window 26 including one or more of the micro image 
regions 310, 314, 316. FIG. 4 is a view of a macro image 
of an actual breast cancer specimen displayed at 1.25X as 
seen on a computer monitor. FIG. 5 is a view of the grid 

10 portion of FIG. 4 outlining a region of interest selected 
by a pathologist displayed at 40X magnification. 

Recall that region A in FIG. 1A was about 4.8mm 
by 3.5mm. This area creates 752 by 480 pixels of sensed 
data, or 360,930 pixels of information. Each pixel sends 

15 information about its location and the image it sensed to 
the computer. The computer stores this information in a 
series of data files (typically .bmp format, but .tif or 
.gif could also be used) . Thus, it can be seen that 
several more pixels of sensed data are available for 

20 viewing on a computer monitor operating at 640 by 480. 
To view the entire image, the user must scroll through 
the image tiles.- However, scrolling need not be done on 
a tile, by tile basis. Rather, the user scrolls by 
pointing to a pixel on the monitor. 

25 Figure 6 is a block diagram showing how the 

control program locates and scrolls through the stored 
image tiles. Using the example from Figure la, a 
complete data structure has been created. When the user 
loads the data structure (of the microscope slide) into 

30 his personal computer or views it from an Internet 

browser, the control program recreates a bit map of the 
stored data. The bit map of the entire slide is shown in 
Figure 6. Image tile A is also high- lighted. This bit 
map enables a user to point to or otherwise reference a 

35 location on the slide. 

The X-Y coordinate information specified in the 
data structure enables X-Y translation of the specific 
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image tiles and specific pixels within the image tile. 
When the control program first loads the image, because 
this image file is so large, only a small number of the 
available tiles are displayed in the active window on the 
5 user's monitor. The user uses his mouse or pointing 
device to scroll through the active window to view the 
entire macro image. The X-Y coordinate information 
selected by the mouse translates into specific image 
tiles or portions therein. The computer takes the mouse 

10 pointer information and retrieves the image data from the 
series of stored tile images and displays them on the 
monitor for viewing the by user. 

Because of the large amount of CCD pixel 
information stored, actual CCD pixel information can be 

15 recreated in the viewing window. The entire system 
operates in a loop, where the user inputs a mouse 
location, the computer translates the mouse location from 
the screen coordinates (screen pixels) to the X-Y 
coordinates on the bit map. 

20 Similarly, the user may select the high 

magnification data images. These are outlined by a dark 
grid, indicating the areas stored. The' user operates the 
mouse in the same manner as described above. The control 
program locates the stored X-Y coordinates and retrieves 

25 the selected parts of the image, CCD stored pixel by CCD 
stored pixel. 

As mentioned above, to save storage space, 
computer 32 can perform a data compression on each of the 
image tile files. A preferred data compression is JPEG, 

30 which is readily transferred and recognized by most 

Internet browser programs. Also, JPEG allows flexibility 
in the amount of data to be compressed, from 20 to 80 
percent. FIG. 8 is file listing such as would be seen 
under Windows 95 file manager showing the data files 

35 included in an alternate data structure, one in which the 
data files have been compressed or converted to JPEG 
(.jpg) format for a breast cancer specimen. The file 
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index.html (shown in Table 3) is the listing which 
contains the X-Y coordinate information for these data 
files. This is the information that is read by the 
dynamic, self -executing program for viewing, 
5 reconstructing and manipulating the image tiles into the 
macro and micro views. 
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TABLE 3 -- index.html 
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Ref erring now to the drawings, and especially to 
FIGS. 9A, 9B and 10, apparatus for synthesizing low 
magnification and high magnification microscopic images 
is shown therein and generally identified by reference 
5 numeral 10. The system includes a computer 12 which is a 
dual Pentium Pro personal computer in combination with a 
Hitachi HV-C20 video camera 14 associated with a Zeiss 
Axioplan 2 microscope 16. The computer system 12 is able 
to receive signals from the camera 14 which captures 

10 light from the microscope 16 having a microscope slide 18 
positioned on an LUDL encoded motorized stage 20. The 
encoded motorized stage 20 includes a MAC 2000 stage 
controller for controlling the stage in response to the 
computer 12. A microscope slide 18 includes a biological 

15 specimen 21 which is to be viewed by the microscope and 
whose image is to be digitized both at low magnification 
and at high magnification as selected by a user. The low 
magnification digitized image is then displayed on a 21 
inch Iiyama video display monitor 22 having resolution of 

20 1600 by 1200 to provide display screens of the type shown 
in FIGS. 1 through 3 including a low magnification image 
24, for instance, at 1.25 power, a high magnification 
image 26, for instance at 40X power and a control window 
or image 28. The low magnification image may have 

25 identified therein a region 30 which is reproduced at 

high magnification in high magnification screen or window 
26 so that a pathologist or other operator of the system 
can review architectural regions of interest in low 
magnification image 24 and simultaneously view them in 

3 0 high magnification in the high magnification screen or 
window 26 to determine whether the cells forming a 
portion of the architectural feature need be examined 
further for cancer or the like or not. 

The computer 10 is constructed around a PCI 

3 5 system bus 40 and has a first Pentium Pro microprocessor 
4 2 and a second pentium pro microprocessor 44 connected 
thereto. The system bus 40 has connected to it a PCI bus 
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50 and an ISA bus 52. The PCI bus 50 has a SCSI 
controller 6 0 connected thereto to send and receive 
information from a hard disk 62. The hard disk 62 also 
is coupled in daisy chain SCSI fashion to a high capacity 
5 removal disk and to a CD Rom drive 66. The hard disks 62 
contains the programs for operating the system for 
controlling the microscope 16 and for processing the 
images as well as for doing a quantitative analysis of 
the selected portions of the histological specimens being 

10 viewed on the slide 18. The system bus 40 also has 

connected to it a random access memory 70 within which 
portions of the program being executed are stored as well 
as a read only memory 72 for holding a bootstrap loader 
as well as portions of the basic input/output operating 

15 system. A floppy disk controller 74 is coupled to the 

system bus 4 0 and has connected to it a floppy disk drive 
76 for reading and writing information to a floppy disk 
as appropriate. A mouse controller 80 is coupled to the 
system bus and has a mouse 82 which operates as a 

20 pointing device for controlling manipulations on the 
screen 22 and within the windows 24, 26 and 28. A 
keyboard controller 90 is connected to the system bus and 
has a keyboard 92 connected thereto. The keyboard 92 may 
be used to send and receive alpha numeric signals to 

25 other portions of the computer. An audio controller 100 
has a plurality of speakers 102 and a microphone 104 
connected thereto for audio input and output and is 
coupled to the system bus 40. A network interface, such 
as a network interface card 104, is connected to the 

30 system bus and can provide signals via a channel 106 to 
other portions of a network or internet to which the 
system may be connected. Likewise, signals can be sent 
out of the system through a modem 110 connected to the 
ISA bus 52 and may be sent via a channel 112, for 

35 instance, to the internet. A printer 116 is connected 
via a parallel I/O controller 118 to the system bus in 
order to provide printouts as appropriate of screens and 
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other information as it is generated. A serial I/O 
controller 122 is connected to the system bus and has 
connected to it a camera controller 124 which is coupled 
to CCD sensors 126 in the cameras. The CCD sensors 126 
5 supply pixel or image signals representative of what is 
found on the slide 18 to an Epix pixci image acquisition 
controller 130 coupled to the PCI bus 50. 

The microscope 16 includes a base 14 0 having a 
stage 20 positioned thereon as well as an objective 

10 turret 142 having a plurality of objectives 144, 146 and 
148 thereon. The objective 144, for instance, may be of 
1.25x objective. The objective 146 may be a 20X 
objective. The objective 148 may be a 4 OX objective. 
Signals from the camera sensors and controller are 

15 supplied over a bus 128 to the image acquisition system 
where they are digitized and supplied to the PCI bus for 
storage in RAM or for backing storage on the hard disk 
62 . 

When a specimen is on the slide 18 the stage 20 

2 0 may be manipulated under the control of the computer 

through a stage controller 16 0 coupled to the serial I/O 
controller 122. Likewise, a microscope controller 162 
controls aspects of the microscope such as the 
illumination, the color temperature or spectral output of 

25 a lamp 168 and the like. For instance, in normal 
operation, when a specimen is placed on the slide, 
specimen slide 18 is placed on the stage 20 in a step 
200, as shown in FIG. 14, the processors 42 or 44 send a 
command through the system bus to cause the serial I/O 

30 controller 122 to signal the microscope controller to 
change, magnification to 1.25X in a step 202. This is 
done by rotating the objective turret of the Axioplan 2 
microscope to select the objective 144. Likewise, the 
controller sets the color temperature of the lamp 168, 

35 sets a pair of neutral density filter wheels 170 and 172 
and sets a field diaphragm 174 for the correct 
illumination. A condenser diaphragm 176 is also 
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controlled and a color filter wheel 180 may also be 
controlled to apply the appropriate filter color to the 
CCD censors 12 6 in the camera. The entire slide is then 
scanned in a step 2 04. The images are tiled and melded 
5 together into the overall image 24 supplied on the screen 
22 to provide the operator in the step 206 with a 
visually inspectable macro image of relevant regions of 
the slide of interest. 

In order to provide the magnified image, the 

10 mouse may be moved to identify a marker segment or region 
which, for instance, may be a rectangular region which 
will cause the microscope to change magnification as at 
step 208 to 4x, 20x, 40x, etc., by rotating the turret to 
bring the appropriate objective lens system into viewing 

15 position. 

Next the user, in a step 209a, uses the mouse to 
select the region on the macro image in order to select 
the micro image to be viewed on the screen 22. In a step 
209b a test is made to determine whether the user has 

20 commanded continued inspection. If the user has, a test 
is made in a step 209c to determine if the magnification 
is to be changed by changing the selected objective. In 
the event the magnification is to be changed control is 
transferred to the step 208. If the magnification is to 

25 remain unchanged control is transferred to the step 209a. 
In the event inspection is not to continue the region 
selected is outlined for higher magnification scan in a 
step 2 09d. In a step 209e, a command may be received to 
scan or acquire the higher magnification image for 

30 display in screen 26. The image may then be archived for 
later analysis, displayed or analyzed immediately. 

In order to perform the magnification called for 
in step 208, the overall illumination and control of the 
microscope will be controlled so that in a step 210 the 

35 objective turret 142 will be rotated to place the higher 
power objective above the slide 18. In a step 212 
voltage to the lamp will be changed to adjust the lamp 
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168 to provide the proper illumination and color 
temperature as predetermined for the selected objective. 
In a step 214, the condenser diaphragm 176 will have its 
opening selected as appropriate to provide the proper 
5 illumination for that objective. In a step 216 , the 

filter turret 180 will select the proper light wavelength 
filter to be supplied to the camera sensors. For 
instance, a red, blue or green filter, as appropriate, 
particularly if the specimen has been stained. In a step 

10 218 the field diaphragm 174 will have its opening 

changed. In a step 220 the neutral density filter wheel 
170 will select a neutral density filter and in a step 
222 the neutral density filter wheel 172 will also select 
a neutral density filter. In a step 224 the X, Y and Z 

15 offsets will be used for reconstruction of the recorded 
image at the magnification and in a step 226 the current 
position will be read from encoders in the stage which 
are accurate to .10 micron. 

In order to identify the selected region the 

20 mouse is moved to that area of the region in a pointing 
operation in a step 24 0 as shown in FIG. 14. The mouse 
may be moved to draw a box around the region selected. 
In a step 242 the X and Y screen points are computed for 
the edges of the regions selected and the computed image 

25 or pixel points are translated to stage coordinate points 
in order to control the stage of the microscope. In a 
step 244 a list of all of the X fields for positioning 
the stage for the objective is stored in random access 
memory and may be backed up on the hard disk. The 

3 0 information from the X offsets for the objective and the 
stage offsets is used as well as the size of the field to 
position the slide properly under the objective to 
capture the micro image . 

When the slide has been positioned properly, as 

35 shown in FIG. 15 in a step 250 the stage is positioned 
for each of the X and Y coordinate values in stage 
coordinate values and the digitized image is captured by 
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the cameras and stored in RAM and backed up on the hard 
disk. The image may be then analyzed quantitatively in 
various manners such as those set forth in the 
previously- identified United States application. 
5 Optionally the image may be stored for archival purposes 
in a step 254 . 

In order to override the specific control 
functions that take place as shown in FIG. 12, a screen 
is provided as shown in FIG. 13 wherein the X-Y step size 

10 can be edited, the X, Y and Z offset can be edited, the 
lamp voltage can be selected, the neutral density filter 
can be selected as well as the opening of the field 
diaphragm and several other microscopic characteristics. 
FIG. 13 is a view of the settings of the microscope 

15 objective properties of the Axioplan 2, computer- 
controlled microscope. 

The X and Y positioning is specifically carried 
out as shown in FIG. 16 where the slide 18 is shown with 
a slide boundary 270, 272, 274 and 276. Stage boundary 

20 for limits of the stage travel for purposes of the stage 
the stage can be moved all the way from an upper left 
hand corner of travel 276 to a lower right hand corner of 
travel 280. At the upper left hand bounded corner of 
travel 278 limits which a signal that the end of travel 

25 has been reached and the stage is then translated a short 
distance 282 in the extra action and a short distance 2 84 
in the Y direction to define the first tile 288 in terms 
of a reference point 290 at its upper left hand corner. 
Since the size of the macro image tile 288 is known, the 

30 next macro image tile 292 may be placed contiguous with 
it by moving the stage appropriately and by measuring the 
location of the stage from the stage in counters without 
the necessity of performing any image manipulation. The 
image tiles 288 and 292 may be abutted without any 

35 substantial overlap or they may be overlapped slightly, 
such as a one pixel with overlap, which is negligible 
insofar as blurring of any adjacent edges of abutted 
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image tiles. The upper left hand corner 300 of the tile 
292 defines the rest of 292 and other tiles can be so 
defined. Micro image tiles can likewise be defined so 
that they are contiguous but not substantially 
5 overlapping, as would interfere with the composite image. 
This avoids the problems encountered with having to 
perform extended computations on digital images in a 
frame storer or multiple frame storage in order to match 
or bring the images into contiguity without blurriness at 

10 the edges of contiguous image tiles. It may be 

appreciated that the low power image 24 has a plurality 
of micro images defined therein which are tiled and which 
are shown in higher magnification as individual tiles 
312, 314, 316 and the like. In addition, the region 310 

15 when magnified as shown in the window 26 may exceed the 
bounds of the window and thus the window may include 
scroll bars or other means for allowing the image 310 
which is larger than the window 26 to be examined from 
within the window 26. 

20 The stage 200 is best seen in FIG. 16A and 

includes the X and Y stepper motors 279 and 281 with 
their respective encoders, which provide a closed loop 
system to give the .1 micron accuracy versus the usual 5 
or 6 micron accuracy of most microscope stages without a 

25 closed loop system. This closed loop system and this 

very high accuracy allow the abutting of the tile images 
for both high magnification and low magnification images 
without the substantial overlap and the time-consuming 
and expensive software currently used to eliminate the 

3 0 overlap and blurriness at the overlapping edges of 
adjacent image tiles. With the precisely positioned 
stage and by using the tiling system described in 
connection with FIG. 16, where the slide is precisely 
positioned relative to a center point CP for the slide, 

35 and the known position of point 278 is always taken from 
the same point, the tiles may be positioned precisely in 
a horizontal row and precisely in vertical rows to 
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reconstruct the macro image and the micro image. This 
reconstruction is done without the use, as in the prior 
art, of extensive software manipulation to eliminate 
overlapping image tiles, horizontally or vertically or 
5 the haphazard orientation of image tiles. 

The present invention also includes the facility 
for allowing remote observation to occur by being able to 
couple the system either over a network communication 
facility to an intranet, for instance via the network 

10 interface, or via a modem or other suitable connection, 
to an internet so that once the image has been scanned 
and stored in memory on hard disks or other storage, 
remote users may be able to access the low magnification 
image as well as the high magnification image and move 

15 around within both images to make determinations as to 
the histological characteristics of the samples. 

An additional feature of the system includes a 
plurality of networked workstations coupled to a first 
computer console 12 having a display screen 22 connected 

20 to the microscope 14. Satellite work stations 350 and 
352 are substantially identical to the work station 12 
including respective computers 354 and 356 coupled to 
displays 358 and 360. The devices can be manipulated 
through input devices 360 and 362 which may include a 

25 keyboard, mouse and the like. Also a third device can be 
connected including a work station 370, having a display 
372, a computer 374 and an input device 376. Each of the 
devices is connected over respective network lines 380, 
382, 384 to the computer 12 which transmission may be via 

30 either net or the like. Each of the different operators 
at the physically separate viewing stations can locate 
regions from the view of entire tissue cross sections via 
a macro view and label the regions for subsequent 
scanning and/or quantitative analysis. A single operator 

35 at the instrument station 12 can locate regions to view 
the entire tissue cross section. Those regions can be 
labeled for subsequent scanning and/or quantitative 
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analysis with subsequent review and physically remote 
viewing stations, for instance, in an operating room or 
in individual pathologists' signout areas in order to 
review analysis results while still maintaining and 
5 reviewing the entire macro view of the tissue and/or the 
individual stored images from which the quantitative 
results were obtained. The viewing stations 350, 352 and 
3 70 can comprise desk top computers, laptops, etc. There 
is no need for a microscope at the network stations 350, 
10 352 and 370. 

In a still further alternative embodiment, 
remote workstations 400, 402, 404, 406 and 408 may be 
connected through a server 410 which may be supplied via 
a packet switched network. The server 410 and may be a 

15 hypertext transport protocol based server of the type 
used for the World Wide Web or may be a telnet type 
server as used previously in internet remote operation 
applications. The server 410 communicates via a 
communications channel 414 with a local computer 416 

20 having a display 418 associated therewith, the local 

computer 416 being connected to the microscope 420, Each 
of the remote work stations 400, 402, 404, 406 and 408 
may perform the same operations as the stations 350, 352 
and 370 although they do it from nearby buildings or even 

25 from around the world, thus providing additional 
flexibility for others to make use of the specimen 
obtained and being viewed under the microscope 420. In 
addition, stored images may be disseminated through the 
server 410 to the remote servers 400 through 408 for 

3 0 further analysis and review. 

While there has been illustrated and described a 
particular embodiment of the present invention, it will 
be appreciated that numerous changes and modifications 
will occur to those skilled in the art, and it is 

35 intended in the appended claims to cover all those 
changes and modifications which followed in the true 
spirit and scope of the present invention. 
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What Is Claimed Is : 

1. A data structure of images taken from a 
specimen on a microscope slide comprising: 

a first series of contiguous, multiple images at 
5 a first magnification abutted against each other to 
create an overall low resolution view of several 
adjacent, original microscope images assembled together; 

the first series of multiple images taken from 
at least a portion of a specimen on the slide; 
10 a second series of contiguous, multiple images 

at a second higher magnification abutted against each 
other to create a high resolution view of several 
adjacent, original microscope images assembled together, 
taken from said portion of the specimen; and 
15 the first and second series of images providing 

multiple resolution images of the slide specimen to a 
viewer. 



2 . A data structure in accordance with Claim 1 
wherein each of the series of contiguous, multiple images 

20 corresponds to an optical image seen by a person through 
an objective lens of a microscope. * 

3 . A data structure in accordance with Claim 1 
wherein each of the series of contiguous, multiple images 
comprises a compressed image using a reduced percentage 

25 of a corresponding original microscope image. 

4 . A data structure in accordance with Claim 1 
wherein an addressable coordinate system is provided for 
the first and second magnification images so that the 
higher magnification images can be easily located with 

30 respect to the lower magnification images. 

5. The data structure of Claim 4 wherein each 
of the image tiles in the first series and in the second 
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series includes coordinate information for enabling 
reconstruction of the entire image. 

6 . The data structure of Claim 4 further 
comprising a dynamic, self -executing program for viewing, 

5 manipulating and reconstructing the image tiles to form 
the low and high resolution images. 

7 . The data structure of Claim 5 further 
comprising a dynamic, self -executing program for viewing, 
manipulating and reconstructing the image tiles to form 

10 the low and high resolution images. 

8 . The data structure of Claim 7 wherein said 
self -executing program comprises a Java applet. 

9. The data structure of Claim 7 wherein said 
self -executing program comprises an Active-X applet. 

15 10. The data structure of Claim 7 wherein said 

self -executing program comprises an Internet web browser. 

11. The data structure of Claim 7 wherein said 
self -executing program comprises an intranet browser. 

12. The data structure of Claim 7 further 

2 0 comprising means for scrolling through said first digital 
image wherein selection of a point and a region on said 
first digital image displays a corresponding image at 
said second magnification in said second digital image. 

13 . The data structure of Claim 7 wherein said 
25 program includes means for displaying the coordinates of 

a point on said first image and said second image. 



14 . A storage medium having digitized images of 
a specimen on a microscope slide comprising: 
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a storage medium; 

a first series of contiguous, multiple images at 
a first magnification abutted against each other to 
create an overall low resolution view of several 
5 adjacent, original microscope images assembled together; 

the first series of multiple images taken from 
at least a portion of a specimen on the slide; 

a second series of contiguous, multiple images 
at a second higher magnification abutted against each 
10 other to create a high resolution view of several 

adjacent, original microscope images assembled together, 
taken from said portion of the specimen; and 

the first and second series of images providing 
multiple resolution images of the slide specimen to a 
15 viewer. 

15. A storage medium in accordance with Claim 
14 wherein each of the series of contiguous, multiple 
images corresponds to an optical image seen by a person 
through an objective lens of a microscope. 

20 16. A storage medium in accordance with Claim 

14 wherein each of the series of contiguous, multiple 
images comprises a compressed image using a reduced 
percentage of a corresponding original microscope image . 

17. A storage medium in accordance with Claim 
25 14 wherein an addressable coordinate system is provided 
for the first and second magnification images so that the 
higher magnification images can be easily located with 
respect to the lower magnification images. 



18. The storage medium of Claim 14 wherein each 
30 of the image tiles in the first series and in the second 
series includes coordinate information for enabling 
reconstruction of the entire image. 
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19. The storage medium of Claim 14 further 
comprising a dynamic, self -executing program for viewing, 
manipulating and reconstructing the image tiles. 

20. The storage medium of Claim 14 further 

5 comprising a dynamic, self -executing program for viewing, 
manipulating and reconstructing the image tiles. 



21. The storage medium of Claim 20 wherein said 
self -executing program comprises a Java applet. 

22. The storage medium of Claim 20 wherein said 
10 self -executing program comprises an Active-X applet. 

23. The storage medium of Claim 20 wherein said 
self -executing program comprises an Internet web browser. 



24. The storage medium of Claim 20 wherein said 
self -executing program comprises an intranet browser. 

15 25. The storage medium of Claim 20 further 

comprising means for scrolling through said first digital 
image wherein selection of a point and a region on said 
first digital image displays a corresponding image at 
said second magnification in said second digital image. 

20 26. The storage medium of Claim 20 wherein said 

program includes means for displaying the coordinates of 
a point on said first image and said second image. 

27. The storage medium of Claim 14 wherein the 
storage medium is selected from the group of CD-rom disks 
25 and Jazz drive disks. 



28. A method of constructing a data structure 
taken from a specimen on a microscope slide comprising 
the steps of: 
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digitally scanning and storing a first series of 
digitized images taken from a portion of a specimen on a 
microscope slide in a first series of contiguous image 
tiles at a first magnification to allow formation of an 
5 overall view; 

digitally scanning and storing a second series 
of digitized images taken from the portion of the 
specimen on the microscope slide in a second series of 
contiguous image tiles at a second, higher magnification; 
10 and 

providing the data structure with the first and 
second series of digitized, stored images to provide a 
user with multiple resolution images from the specimen. 

29. The method of Claim 28 further comprising 
15 the step of : 

providing the data structure with a dynamic, 
self -executing program effective for viewing, 
manipulating and reconstructing the image tiles. 

30. The method of Claim 28 further comprising 
20 the step of : 

providing each of the image tiles in the first 
series and in the second series with coordinate 
information for enabling reconstruction of the first and 
second images. 

25 31. The method of Claim 30 wherein said 

self-executing program comprises the step of providing a 
Java applet. 



32. The method of Claim 30 further comprising 
the step of displaying the X-Y coordinates of a selected 
30 point on the first and second image. 
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33. The method of Claim 28 further comprising 
the steps of compressing the digitized images and storing 
the compressed digitized images. 

34. The method of Claim 28 wherein the step of 
5 digitizing and storing the images comprises: 

using a higher resolution central portion of 
optical images taken through the microscope; and 

discarding fuzzy outer portions of the optical 

image . 

10 35, The method of Claim 28 wherein each of the 

digitized images of the series corresponds to 
substantially an optical view as seen by a person through 
an objective lens of the microscope. 



36. A storage medium having digitized images 
15 from a specimen on a piece on a microscopic support 

comprising : 

a storage medium; 

a first collection of digitized image fields of 
view at a first magnification coherently stored on the 
20 storage medium to provide an overall low resolution view 
from original microscope images of the specimen on the 
slide taken at a first resolution; 

a collection of digitized image fields of view 
at a higher magnification coherently stored on the 
25 storage medium to provide a higher resolution image for 
viewing from a selected portion of the overall view; and 

a control program stored with the first and 
second collection of digitized images fields to allow a 
user to move back and forth between the overall view at 
3 0 the first lower resolution and the selected higher 
resolution images. 

37. A storage medium in accordance with Claim 
36 wherein the first collection of single image fields 
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comprises a collection of tiled images abutted one 
against the other. 

38. A storage medium in accordance with Claim 

37 wherein one tile is one picture viewed through the 
5 microscope. 

39. A storage medium in accordance with Claim 

38 wherein the second collection of image fields 
comprises at least three higher resolution image fields 
of view, each having a substantially different resolution 

10 and each capable of being selected by a user. 

40. A storage medium in accordance with Claim 

3 9 wherein the storage medium is on a web browser and the 
view is accessing the browser for viewing the specimen's 
digitized images, 

15 41. A storage medium in accordance with Claim 

4 0 wherein the stored, digitized images are JPEG images 
received over the Internet; and 

the stored control program comprises a HTML file 
and an active, self -executing program. 

2 0 42 . A storage medium in accordance with Claim 

36 wherein the storage medium is a CD-Rom. 

43 . A storage medium in accordance with Claim 
41 wherein the active, self -executing program comprises a 
Java applet . 

25 44 . A method of constructing and using a self- 

executing data structure of an image of a specimen on a 
microscope slide comprising: 

digitally scanning and storing multiple 
magnification and multiple resolution images from the 

30 specimen on a microscope slide, to create a plurality of 
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individual image tiles having multiple magnifications and 
multiple resolutions; 

providing a dynamic, self -executing program on 
the data structure for viewing, manipulating and 
5 reconstructing the image tiles; and 

transferring the scanned, digital image tiles 
with the dynamic, self -executing program to allow viewing 
of a digital image of substantially larger image area 
than the area of the individually acquired tiles and at 
10 multiple resolutions. 

45. The method of Claim 44 further comprising 
the steps of : 

displaying a first image comprising a portion of 
the specimen as an overall macro view; and 
15 displaying a second image comprising higher 

resolution view from the specimen on the microscope slide 
at a higher magnification than, the magnification of the 
overall macro view. 

46. The method of Claim 45 further comprising: 
20 selecting a point on said overall macro image 

with a marker; and 

producing a corresponding higher magnification 
image at the location of the marker. 

47. The method of Claim 44 further comprising 
25 displaying the X-Y coordinates of said point. 

48. The method of Claim 44 wherein said image 
tiles are stored as bit-mapped files. 

49. The method of Claim 48 further comprising 
converting said bit-mapped files to JPEG files. 
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50. The method of Claim 44, comprising scanning 
and storing said images of said specimen at at least 
three magnifications. 



51. A method in accordance with Claim 44 

5 including the step of transferring the digital images to 
a web browser. 

52. A method in accordance with Claim 51 
including the step of transferring of the digital images 
over a dynamic Internet . 

10 53. A method in accordance with Claim 51 

including the step of transferring the digital images 
over a dynamic intranet. 



54. A method in accordance with Claim 51 

including the step of re toggling between lower 

15 magnification images and higher magnification images 

stored on the web browser. 



55. A method in accordance with Claim 46 
wherein the step of selecting a point includes: 

moving across tile boundaries to a desired point 
20 on a selected macro tile image; and 

executing a command to product the micro image 
from the selected macro tile image point . 

56. A method in accordance with Claim 44 
including the step of : 

25 scrolling a portion of an image being viewed in 

a direction to cause the image being viewed to shift to 
include, in a new image, a portion of the image from a 
neighboring tiled image that was not previously viewed in 
the last image viewed by a user. 
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57. A program for creating a data structure of 
an image of a specimen on a microscope slide comprising: 

a scanning routine for digitally scanning the 
image of the specimen on a microscope slide at a 
5 plurality of image magnifications; 

a recording routine for recording the scanned 
digital image in a series of contiguous image tiles; and 

a linking routine for linking the series of 
contiguous image tiles with a dynamic, self -executing 
10 program effective for viewing, manipulating and 
reconstructing the image tiles. 

58. The program of Claim 57 further comprising 
a first display routine for displaying a micro image 
comprising a portion of the scanned image at a first 

15 magnification and a second display routine for displaying 
a macro second image comprising an overall view from the 
specimen on the microscope slide. 

59. The program of Claim 57 further comprising 
a routine for selecting a point on said macro image and 

20 for producing a corresponding micro image at said point. 

60. The program of Claim 57 further comprising 
a coordinate display routine for displaying the 
coordinates of said point to the user. 

61. An apparatus for creating a data structure 
25 comprising: 

a computer-controlled microscope imaging system 
for digitally scanning images from a specimen on a 
microscope support at a plurality of image 
magnifications ; 
3 0 a program for recording the scanned digital 

images in a series of contiguous image tiles; and 

a program for linking the series of contiguous 
image tiles with a dynamic, self -executing program 
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effective for viewing, manipulating and reconstructing 
the image tiles. 

62. An apparatus is accordance with Claim 61 
wherein a data compressor compresses the data of the 

5 digitally scanned significantly to allow the scanned 
digitized images to be sent over the Internet. 

63. An apparatus in accordance with Claim 61 
wherein an addressable coordinate system provides 
addresses so that images can be seamed together and 

10 higher magnification images can be easily located with 
respect to the lower magnification images. 

64. An apparatus in accordance with Claim 61 
wherein a program for scrolling allows the user to scroll 
a portion of a neighboring image into view. 

15 65. An apparatus in accordance with Claim 61 

including an address display to display the coordinates 
to assist multiple viewers to identify the same area for 
analysis and commentary. 

66. An apparatus in accordance with Claim 61 
20 further comprising: 

a browser for storing the dynamic, self- 
executing program; and 

a monitor for viewing the images stored on the 
monitor and for flipping back and forth between low 
25 resolution macro images and high resolution micro images. 

67. An apparatus in accordance with Claim 66 
including a marker program to mark an addressable area on 
the. macro image and to cause the addressed area to appear 
at a higher resolution, micro image on the monitor. 
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Fig. 7a 

Uncompressed Data Files in BMP format 
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